Understanding Implicit Regularization in Over-Parameterized Single Index Model

In this paper, we leverage over-parameterization to design regularization-free algorithms for the high-dimensional single index model and provide theoretical guarantees induced implicit regularization phenomenon. Specifically, study both vector matrix models where link function is nonlinear unknown, signal parameter either a sparse or low-rank symmetric matrix, response variable can be heavy-tailed. To gain better understanding of role played by without excess technicality, assume that distribution covariates known priori. For settings, construct an over-parameterized least-squares loss employing score transform robust truncation step designed specifically heavy-tailed data. We propose estimate true applying gradient descent function. When initialization close origin stepsize sufficiently small, prove obtained solution achieves minimax optimal statistical rates convergence in cases. addition, our experimental results support findings also demonstrate methods empirically outperform classical with explicit terms $\ell_2$-statistical rate selection consistency.